Conversation
A raw heap output built by Planner is stored as a WeightedFrequencyV1 frame (sketchlib WeightedFrequency bytes, persisted with its own encoding tag). Stored heap readout decodes such frames for their catalog family, merges them, and ranks items under the legacy heap key (a canonical label identity renders as its series key). Point-count readouts over these frames are reported unsupported rather than guessed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CMS/CountSketch heaps now take the same Planner execution as every other raw output, so the backend's per-sample item and weight interpreter is deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mpty state Revision views and derived publications choose a stored frame's encoding from its state, so a Planner heap is never re-tagged as a legacy msgpack heap; carry-in checks treat it as a full frame. Heap frames decode only for their catalog's matrix shape, identity items render as series keys only when they name `__name__`, and a heap's empty state is a Planner heap, so revisions install for heap outputs. Heap items that omit identity labels are rejected at install until readout can name their series. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #797.
Why
After #796/#797 every raw output except CMS/CountSketch heaps executes Planner's precompute graph. Heaps stayed on the backend's per-sample kernel (the last item/weight interpreter in
raw_dag.rs) because stored heap readout could only decode the legacy integer heap kernel, not Planner's weighted-frequency state.What
WeightedFrequencyV1frame (sketchlibWeightedFrequencybytes, persisted with encoding tag 6). One helper picks a frame's encoding from its state for live appends, revision views and derived publications; carry-in checks treat it as a full frame.SummaryState::WeightedFrequencydecodes such frames for their catalog family and matrix shape, merges them, and ranks items under the legacy heap key (;-joined parts; a canonical series identity renders as its series key). PointCount/total over these frames is reported unsupported.Before this PR
topk(3, count_over_time(top_endpoint_qps[5s]))with a CMS heap: each sample was keyed by the backend (series keystring) and added to aCountMinSketchWithHeapAccumulator.After this PR
The same output is built by Planner's
KeyedSummaryBuildover the pane batch (item =LabelIdentityof the series' label set);registered_temporal_topk_cms_heap/_count_sketch_heapreturn the same ranked items (top_endpoint_qps{endpoint="gamma"}= 200, …) and range results as before.Behaviour differences
Float64weights; the legacy CMS heap rounded each weight to an integer count.WeightedFrequencyV1frames; a range mixing them with legacy heap frames of the same output fails with a family mismatch (a new plan generation starts new outputs). Older binaries do not read tag 6.topk by (…)identity items) are rejected at install until readout can restore their group labels.Remaining (backend computation still present)
RawDagProgram::empty_stateconstructs empty family states (heaps via Planner's constructor; other families via the backend kernel factory) for known-empty counter panes and the revision codec check.PrecomputeMaterializationcomputation fields (aggregation_type,parameters,spatial_filter,aggregated_labels,sample_update_rule) are still cross-checked against the DAG at install; removing them changes plan identities.merge_with, and late admitted corrections are coalesced withmerge_with(both surface errors).Validation
cargo fmt --all -- --check;cargo clippy --workspace --all-targets --locked -- -D warnings;cargo test --workspace --locked --lib;cargo test -p data_plane --locked --test asapquery_compatibility_process_e2e -- --test-threads=1(26 passed, incl. both registered heap TopK tests);cargo test --workspace --locked --no-fail-fast(every test target) passes at the top of the stack. New tests: heap frame decode/merge/rank and shape rejection, heap empty state is a Planner heap frame. Reviewed by a separate reviewer agent; findings addressed in the last commit.🤖 Generated with Claude Code